{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Online serving\n",
    "\n",
    "<div align=\"left\">\n",
    "<a target=\"_blank\" href=\"https://console.anyscale.com/\"><img src=\"https://raw.githubusercontent.com/ray-project/ray/c34b74c22a9390aa89baf80815ede59397786d2e/doc/source/_static/img/run-on-anyscale.svg\"></a>&nbsp;\n",
    "\n",
    "<a href=\"https://github.com/anyscale/multimodal-ai\" role=\"button\"><img src=\"https://img.shields.io/static/v1?label=&amp;message=View%20On%20GitHub&amp;color=586069&amp;logo=github&amp;labelColor=2f363d\"></a>&nbsp;\n",
    "</div>\n",
    "\n",
    "This tutorial launches an online service that deploys the trained model to generate predictions and autoscales based on incoming traffic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92mSuccessfully registered `ipywidgets, matplotlib` and 4 other packages to be installed on all cluster nodes.\u001b[0m\n",
      "\u001b[92mView and update dependencies here: https://console.anyscale.com/cld_kvedZWag2qA8i5BjxUevf5i7/prj_cz951f43jjdybtzkx1s5sjgz99/workspaces/expwrk_23ry3pgfn3jgq2jk3e5z25udhz?workspace-tab=dependencies\u001b[0m\n",
      "\u001b[92mSuccessfully registered `doggos` package to be installed on all cluster nodes.\u001b[0m\n",
      "\u001b[92mView and update dependencies here: https://console.anyscale.com/cld_kvedZWag2qA8i5BjxUevf5i7/prj_cz951f43jjdybtzkx1s5sjgz99/workspaces/expwrk_23ry3pgfn3jgq2jk3e5z25udhz?workspace-tab=dependencies\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "pip install -q -r /home/ray/default/requirements.txt\n",
    "pip install -q -e /home/ray/default/doggos\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: A kernel restart may be required for all dependencies to become available. \n",
    "\n",
    "If using **uv**, then:\n",
    "1. Turn off the runtime dependencies (`Dependencies` tab up top > Toggle off `Pip packages`). And no need to run the `pip install` commands above.\n",
    "2. Change the python kernel of this notebook to use the `venv` (Click on `base (Python x.yy.zz)` on top right cordern of notebook > `Select another Kernel` > `Python Environments...` > `Create Python Environment` > `Venv` > `Use Existing`) and done! Now all the notebook's cells will use the virtual env.\n",
    "3. Change the py executable to use `uv run` instead of `python` by adding this line after importing ray.\n",
    "```python\n",
    "import os\n",
    "os.environ.pop(\"RAY_RUNTIME_ENV_HOOK\", None)\n",
    "import ray\n",
    "ray.init(runtime_env={\"py_executable\": \"uv run\", \"working_dir\": \"/home/ray/default\"})\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload all\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import ray\n",
    "import sys\n",
    "sys.path.append(os.path.abspath(\"../doggos/\"))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If using UV\n",
    "# os.environ.pop(\"RAY_RUNTIME_ENV_HOOK\", None)\n",
    "# ray.init(runtime_env={\"py_executable\": \"uv run\", \"working_dir\": \"/home/ray/default\"})\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from fastapi import FastAPI\n",
    "import mlflow\n",
    "import requests\n",
    "from starlette.requests import Request\n",
    "from urllib.parse import urlparse\n",
    "from ray import serve\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from PIL import Image\n",
    "import torch\n",
    "from transformers import CLIPModel, CLIPProcessor\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from doggos.infer import TorchPredictor\n",
    "from doggos.model import collate_fn\n",
    "from doggos.utils import url_to_array\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deployments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First create a deployment for the trained model that generates a probability distribution for a given image URL. You can specify the compute you want to use with `ray_actor_options`, and how you want to horizontally scale, with `num_replicas`, this specific deployment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@serve.deployment(\n",
    "    num_replicas=\"1\", \n",
    "    ray_actor_options={\n",
    "        \"num_gpus\": 1, \n",
    "        \"accelerator_type\": \"T4\",\n",
    "    },\n",
    ")\n",
    "class ClassPredictor:\n",
    "    def __init__(self, model_id, artifacts_dir, device=\"cuda\"):\n",
    "        \"\"\"Initialize the model.\"\"\"\n",
    "        # Embdding model\n",
    "        self.processor = CLIPProcessor.from_pretrained(model_id)\n",
    "        self.model = CLIPModel.from_pretrained(model_id)\n",
    "        self.model.to(device=device)\n",
    "        self.device = device\n",
    "\n",
    "        # Trained classifier\n",
    "        self.predictor = TorchPredictor.from_artifacts_dir(artifacts_dir=artifacts_dir)\n",
    "        self.preprocessor = self.predictor.preprocessor\n",
    "\n",
    "    def get_probabilities(self, url):\n",
    "        image = Image.fromarray(np.uint8(url_to_array(url=url))).convert(\"RGB\")\n",
    "        inputs = self.processor(images=[image], return_tensors=\"pt\", padding=True).to(self.device)\n",
    "        with torch.inference_mode():\n",
    "            embedding = self.model.get_image_features(**inputs).cpu().numpy()\n",
    "        outputs = self.predictor.predict_probabilities(\n",
    "            collate_fn({\"embedding\": embedding}))\n",
    "        return {\"probabilities\": outputs[\"probabilities\"][0]}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert\"> <b>🧱 Model composition</b>\n",
    "\n",
    "Ray Serve makes it easy to do [model composition](https://docs.ray.io/en/latest/serve/model_composition.html) where you can compose multiple deployments containing ML models or business logic into a single application. You can independently scale even fractional resources, and configure each of your deployments.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/anyscale/multimodal-ai/refs/heads/main/images/serve_composition.png\" width=800>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define app.\n",
    "api = FastAPI(\n",
    "    title=\"doggos\", \n",
    "    description=\"classify your dog\", \n",
    "    version=\"0.1\",\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "@serve.deployment\n",
    "@serve.ingress(api)\n",
    "class Doggos:\n",
    "    def __init__(self, classifier):\n",
    "        self.classifier = classifier\n",
    "        \n",
    "    @api.post(\"/predict/\")\n",
    "    async def predict(self, request: Request):\n",
    "        data = await request.json()\n",
    "        probabilities = await self.classifier.get_probabilities.remote(url=data[\"url\"])\n",
    "        return probabilities\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Model registry.\n",
    "model_registry = \"/mnt/cluster_storage/mlflow/doggos\"\n",
    "experiment_name = \"doggos\"\n",
    "mlflow.set_tracking_uri(f\"file:{model_registry}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get best_run's artifact_dir.\n",
    "sorted_runs = mlflow.search_runs(\n",
    "    experiment_names=[experiment_name], \n",
    "    order_by=[\"metrics.val_loss ASC\"])\n",
    "best_run = sorted_runs.iloc[0]\n",
    "artifacts_dir = urlparse(best_run.artifact_uri).path\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define app.\n",
    "app = Doggos.bind(\n",
    "    classifier=ClassPredictor.bind(\n",
    "        model_id=\"openai/clip-vit-base-patch32\",\n",
    "        artifacts_dir=artifacts_dir,\n",
    "        device=\"cuda\"\n",
    "    )\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2025-08-28 05:15:38,455\tINFO worker.py:1771 -- Connecting to existing Ray cluster at address: 10.0.17.148:6379...\n",
      "2025-08-28 05:15:38,465\tINFO worker.py:1942 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32mhttps://session-jhxhj69d6ttkjctcxfnsfe7gwk.i.anyscaleuserdata.com \u001b[39m\u001b[22m\n",
      "2025-08-28 05:15:38,471\tINFO packaging.py:588 -- Creating a file package for local module '/home/ray/default/doggos/doggos'.\n",
      "2025-08-28 05:15:38,475\tINFO packaging.py:380 -- Pushing file package 'gcs://_ray_pkg_62e649352ce105b6.zip' (0.04MiB) to Ray cluster...\n",
      "2025-08-28 05:15:38,476\tINFO packaging.py:393 -- Successfully pushed file package 'gcs://_ray_pkg_62e649352ce105b6.zip'.\n",
      "2025-08-28 05:15:38,478\tINFO packaging.py:380 -- Pushing file package 'gcs://_ray_pkg_c3f5a1927d401ecc93333d17727d37c3401aeed9.zip' (1.08MiB) to Ray cluster...\n",
      "2025-08-28 05:15:38,484\tINFO packaging.py:393 -- Successfully pushed file package 'gcs://_ray_pkg_c3f5a1927d401ecc93333d17727d37c3401aeed9.zip'.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[36m(autoscaler +9s)\u001b[0m Tip: use `ray status` to view detailed cluster status. To disable these messages, set RAY_SCHEDULER_EVENTS=0.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[36m(ProxyActor pid=42150)\u001b[0m INFO 2025-08-28 05:15:42,208 proxy 10.0.17.148 -- Proxy starting on node 524d54fa7a3dfe7fcd55149e6efeaa7a697a4ce87282da72073206b6 (HTTP port: 8000).\n",
      "INFO 2025-08-28 05:15:42,290 serve 41929 -- Started Serve in namespace \"serve\".\n",
      "\u001b[36m(ProxyActor pid=42150)\u001b[0m INFO 2025-08-28 05:15:42,286 proxy 10.0.17.148 -- Got updated endpoints: {}.\n",
      "\u001b[36m(ServeController pid=42086)\u001b[0m INFO 2025-08-28 05:15:47,403 controller 42086 -- Deploying new version of Deployment(name='ClassPredictor', app='default') (initial target replicas: 1).\n",
      "\u001b[36m(ServeController pid=42086)\u001b[0m INFO 2025-08-28 05:15:47,404 controller 42086 -- Deploying new version of Deployment(name='Doggos', app='default') (initial target replicas: 1).\n",
      "\u001b[36m(ProxyActor pid=42150)\u001b[0m INFO 2025-08-28 05:15:47,423 proxy 10.0.17.148 -- Got updated endpoints: {Deployment(name='Doggos', app='default'): EndpointInfo(route='/', app_is_cross_language=False)}.\n",
      "\u001b[36m(ProxyActor pid=42150)\u001b[0m WARNING 2025-08-28 05:15:47,430 proxy 10.0.17.148 -- ANYSCALE_RAY_SERVE_GRPC_RUN_PROXY_ROUTER_SEPARATE_LOOP has been deprecated and will be removed in the ray v2.50.0. Please use RAY_SERVE_RUN_ROUTER_IN_SEPARATE_LOOP instead.\n",
      "\u001b[36m(ProxyActor pid=42150)\u001b[0m INFO 2025-08-28 05:15:47,434 proxy 10.0.17.148 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x79fb040771d0>.\n",
      "\u001b[36m(ServeController pid=42086)\u001b[0m INFO 2025-08-28 05:15:47,524 controller 42086 -- Adding 1 replica to Deployment(name='ClassPredictor', app='default').\n",
      "\u001b[36m(ServeController pid=42086)\u001b[0m INFO 2025-08-28 05:15:47,525 controller 42086 -- Adding 1 replica to Deployment(name='Doggos', app='default').\n",
      "\u001b[36m(ServeReplica:default:ClassPredictor pid=20055, ip=10.0.5.20)\u001b[0m Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.\n",
      "\u001b[36m(ProxyActor pid=20172, ip=10.0.5.20)\u001b[0m INFO 2025-08-28 05:15:56,055 proxy 10.0.5.20 -- Proxy starting on node b84e244dca75c40ea981202cae7a1a06df9598ac29ad2b18e1bedb99 (HTTP port: 8000).\n",
      "\u001b[36m(ProxyActor pid=20172, ip=10.0.5.20)\u001b[0m INFO 2025-08-28 05:15:56,131 proxy 10.0.5.20 -- Got updated endpoints: {Deployment(name='Doggos', app='default'): EndpointInfo(route='/', app_is_cross_language=False)}.\n",
      "\u001b[36m(ProxyActor pid=20172, ip=10.0.5.20)\u001b[0m WARNING 2025-08-28 05:15:56,137 proxy 10.0.5.20 -- ANYSCALE_RAY_SERVE_GRPC_RUN_PROXY_ROUTER_SEPARATE_LOOP has been deprecated and will be removed in the ray v2.50.0. Please use RAY_SERVE_RUN_ROUTER_IN_SEPARATE_LOOP instead.\n",
      "\u001b[36m(ProxyActor pid=20172, ip=10.0.5.20)\u001b[0m INFO 2025-08-28 05:15:56,141 proxy 10.0.5.20 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x76c838542d80>.\n",
      "INFO 2025-08-28 05:15:57,505 serve 41929 -- Application 'default' is ready at http://127.0.0.1:8000/.\n",
      "INFO 2025-08-28 05:15:57,511 serve 41929 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x7955d893cd70>.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DeploymentHandle(deployment='Doggos')"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Run service locally.\n",
    "serve.run(app, route_prefix=\"/\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[36m(ServeReplica:default:Doggos pid=42244)\u001b[0m INFO 2025-08-28 05:15:57,646 default_Doggos fs1weamq 31c15b70-89a9-4b2d-b4ab-f8424fe6d8d2 -- Started <ray.serve._private.router.SharedRouterLongPollClient object at 0x7ba028149fd0>.\n",
      "\u001b[36m(ServeReplica:default:ClassPredictor pid=20055, ip=10.0.5.20)\u001b[0m /home/ray/anaconda3/lib/python3.12/site-packages/ray/serve/_private/replica.py:1397: UserWarning: Calling sync method 'get_probabilities' directly on the asyncio loop. In a future version, sync methods will be run in a threadpool by default. Ensure your sync methods are thread safe or keep the existing behavior by making them `async def`. Opt into the new behavior by setting RAY_SERVE_RUN_SYNC_IN_THREADPOOL=1.\n",
      "\u001b[36m(ServeReplica:default:ClassPredictor pid=20055, ip=10.0.5.20)\u001b[0m   warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('border_collie', 0.1990548074245453),\n",
       " ('collie', 0.1363961398601532),\n",
       " ('german_shepherd', 0.07545585185289383)]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[36m(ServeReplica:default:Doggos pid=42244)\u001b[0m INFO 2025-08-28 05:15:58,150 default_Doggos fs1weamq 31c15b70-89a9-4b2d-b4ab-f8424fe6d8d2 -- POST /predict/ 200 516.2ms\n",
      "\u001b[36m(ServeReplica:default:ClassPredictor pid=20055, ip=10.0.5.20)\u001b[0m INFO 2025-08-28 05:15:58,148 default_ClassPredictor y7tebd3e 31c15b70-89a9-4b2d-b4ab-f8424fe6d8d2 -- CALL /predict/ OK 491.4ms\n"
     ]
    }
   ],
   "source": [
    "# Send a request.\n",
    "url = \"https://doggos-dataset.s3.us-west-2.amazonaws.com/samara.png\"\n",
    "data = {\"url\": url}\n",
    "response = requests.post(\"http://127.0.0.1:8000/predict/\", json=data)\n",
    "probabilities = response.json()[\"probabilities\"]\n",
    "sorted_probabilities = sorted(probabilities.items(), key=lambda x: x[1], reverse=True)\n",
    "sorted_probabilities[0:3]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ray Serve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Ray Serve](https://docs.ray.io/en/latest/serve/index.html) is a highly scalable and flexible model serving library for building online inference APIs that allows you to:\n",
    "- Wrap models and business logic as separate [serve deployments](https://docs.ray.io/en/latest/serve/key-concepts.html#deployment) and [connect](https://docs.ray.io/en/latest/serve/model_composition.html) them together (pipeline, ensemble, etc.)\n",
    "- Avoid one large service that's network and compute bounded and an inefficient use of resources.\n",
    "- Utilize fractional heterogeneous [resources](https://docs.ray.io/en/latest/serve/resource-allocation.html), which **isn't possible** with SageMaker, Vertex, KServe, etc., and horizontally scale with`num_replicas`.\n",
    "- [autoscale](https://docs.ray.io/en/latest/serve/autoscaling-guide.html) up and down based on traffic.\n",
    "- Integrate with [FastAPI and HTTP](https://docs.ray.io/en/latest/serve/http-guide.html).\n",
    "- Set up a [gRPC service](https://docs.ray.io/en/latest/serve/advanced-guides/grpc-guide.html#set-up-a-grpc-service) to build distributed systems and microservices.\n",
    "- Enable [dynamic batching](https://docs.ray.io/en/latest/serve/advanced-guides/dyn-req-batch.html) based on batch size, time, etc.\n",
    "- Access a suite of [utilities for serving LLMs](https://docs.ray.io/en/latest/serve/llm/serving-llms.html) that are inference-engine agnostic and have batteries-included support for LLM-specific features such as multi-LoRA support\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/anyscale/multimodal-ai/refs/heads/main/images/ray_serve.png\" width=500>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🔥 [RayTurbo Serve](https://docs.anyscale.com/rayturbo/rayturbo-serve) on Anyscale has more functionality on top of Ray Serve:\n",
    "- **fast autoscaling and model loading** to get services up and running even faster with [5x improvements](https://www.anyscale.com/blog/autoscale-large-ai-models-faster) even for LLMs.\n",
    "- 54% **higher QPS** and up-to 3x **streaming tokens per second** for high traffic serving use-cases with no proxy bottlenecks.\n",
    "- **replica compaction** into fewer nodes where possible to reduce resource fragmentation and improve hardware utilization.\n",
    "- **zero-downtime** [incremental rollouts](https://docs.anyscale.com/platform/services/update-a-service/#resource-constrained-updates) so your service is never interrupted.\n",
    "- [**different environments**](https://docs.anyscale.com/platform/services/multi-app/#multiple-applications-in-different-containers) for each service in a multi-serve application.\n",
    "- **multi availability-zone** aware scheduling of Ray Serve replicas to provide higher redundancy to availability zone failures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Observability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Ray dashboard and specifically the [Serve view](https://docs.ray.io/en/latest/ray-observability/getting-started.html#serve-view) automatically captures observability for Ray Serve applications. You can view the service [deployments and their replicas](https://docs.ray.io/en/latest/serve/key-concepts.html#serve-key-concepts-deployment) and time-series metrics to see the service's health.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/anyscale/multimodal-ai/refs/heads/main/images/serve_dashboard.png\" width=1000>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Production services"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Anyscale Services](https://docs.anyscale.com/platform/services/) ([API ref](https://docs.anyscale.com/reference/service-api/)) offers a fault tolerant, scalable, and optimized way to serve Ray Serve applications. You can:\n",
    "- [rollout and update](https://docs.anyscale.com/platform/services/update-a-service) services with canary deployment and zero-downtime upgrades.\n",
    "- [monitor](https://docs.anyscale.com/platform/services/monitoring) services through a dedicated service page, unified log viewer, tracing, set up alerts, etc.\n",
    "- scale a service, with `num_replicas=auto`, and utilize replica compaction to consolidate nodes that are fractionally utilized.\n",
    "- get [head node fault tolerance](https://docs.anyscale.com/platform/services/production-best-practices#head-node-ft). OSS Ray recovers from failed workers and replicas but not head node crashes.\n",
    "- serve [multiple applications](https://docs.anyscale.com/platform/services/multi-app) in a single service.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/anyscale/multimodal-ai/refs/heads/main/images/canary.png\" width=1000>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: \n",
    "- This tutorial uses a `containerfile` to define dependencies, but you could easily use a pre-built image as well.\n",
    "- You can specify the compute as a [compute config](https://docs.anyscale.com/configuration/compute-configuration/) or inline in a [Service config](https://docs.anyscale.com/reference/service-api/) file.\n",
    "- When you don't specify compute while launching from a workspace, this configuration defaults to the compute configuration of the workspace."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "# Production online service.\n",
    "anyscale service deploy -f /home/ray/default/configs/service.yaml\n",
    "```\n",
    "```\n",
    "(anyscale +1.9s) Restarting existing service 'doggos-app'.\n",
    "(anyscale +3.2s) Uploading local dir '/home/ray/default' to cloud storage.\n",
    "(anyscale +5.2s) Including workspace-managed pip dependencies.\n",
    "(anyscale +5.8s) Service 'doggos-app' deployed (version ID: akz9ul28).\n",
    "(anyscale +5.8s) View the service in the UI: 'https://console.anyscale.com/services/service2_6hxismeqf1fkd2h7pfmljmncvm'\n",
    "(anyscale +5.8s) Query the service once it's running using the following curl command (add the path you want to query):\n",
    "(anyscale +5.8s) curl -H \"Authorization: Bearer <BEARER_TOKEN>\" https://doggos-app-bxauk.cld-kvedzwag2qa8i5bj.s.anyscaleuserdata.com/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```sh\n",
    "curl -X POST \"https://doggos-app-bxauk.cld-kvedzwag2qa8i5bj.s.anyscaleuserdata.com/predict/\" \\\n",
    "     -H \"Authorization: Bearer <BEARER_TOKEN>\" \\\n",
    "     -H \"Content-Type: application/json\" \\\n",
    "     -d '{\"url\": \"https://doggos-dataset.s3.us-west-2.amazonaws.com/samara.png\", \"k\": 4}'\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "# Terminate service.\n",
    "anyscale service terminate --name doggos-app\n",
    "```\n",
    "```\n",
    "(anyscale +1.5s) Service service2_6hxismeqf1fkd2h7pfmljmncvm terminate initiated.\n",
    "(anyscale +1.5s) View the service in the UI at https://console.anyscale.com/services/service2_6hxismeqf1fkd2h7pfmljmncvm\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CI/CD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While Anyscale [Jobs](https://docs.anyscale.com/platform/jobs/) and [Services](https://docs.anyscale.com/platform/services/) are useful atomic concepts that help you productionize workloads, they're also useful for nodes in a larger ML DAG or [CI/CD workflow](https://docs.anyscale.com/ci-cd/). You can chain Jobs together, store results and then serve your application with those artifacts. From there, you can trigger updates to your service and retrigger the Jobs based on events, time, etc. While you can simply use the Anyscale CLI to integrate with any orchestration platform, Anyscale does support some purpose-built integrations like [Airflow](https://docs.anyscale.com/ci-cd/apache-airflow/) and [Prefect](https://github.com/anyscale/prefect-anyscale).\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/anyscale/multimodal-ai/refs/heads/main/images/cicd.png\" width=700>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**🚨 Note**: Reset this notebook using the **\"🔄 Restart\"** button location at the notebook's menu bar. This way we can free up all the variables, utils, etc. used in this notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
